Image correction apparatus, image correction method and program

ABSTRACT

A transmitting and receiving section receives an image rendered on the basis of a position or an orientation of a head mounted display at a first time together with information regarding the position or the orientation at the first time. A correction section corrects the image by acquiring information regarding a position or an orientation of the head mounted display at a second time of displaying the image, translating or rotationally moving a visual field of the head mounted display in a screen coordinate system in response to a difference between the position or the orientation at the first time and the position or the orientation at the second time, and pasting the image on the translated or rotationally moved visual field as a texture.

TECHNICAL FIELD

The present invention relates to an apparatus and a method forgenerating an image and correcting the image.

BACKGROUND ART

A user wears a head mounted display connected to a video game machine onthe user's head, and plays a game by operating a controller or the likewhile viewing a screen displayed on the head mounted display. Whenwearing the head mounted display, the user does not view anything but avideo displayed on the head mounted display and effects of enhancing asense of immersion into a video world and further improvingentertainment of the game are, therefore, produced. Furthermore, bydisplaying a VR (Virtual Reality) video on the head mounted display,causing the user wearing the head mounted display on the head to rotatethe head, and displaying a virtual space where the user can take a360-degree view, the sense of immersion into the video is furtherenhanced and operability of an application such as the game is improved.

SUMMARY Technical Problems

In a case in which the head mounted display takes on a head trackingfunction and a virtual reality video is generated by changing aviewpoint and a line-of-sight direction to be interlocked with a motionof the user's head, a delay is generated from generation to display ofthe virtual reality video. As a result, a lag is generated between adirection of the user's head premised at a point in time of generatingthe video and a direction of the user's head at a point in time ofdisplaying the video on the head mounted display, and the user oftenfalls into a sense of feeling sick (referred to as “VR Sickness (VirtualRealty Sickness)” or the like).

Furthermore, in a case of streaming delivery of the video displayed onthe head mounted display from a server, a communication delay is addedto the lag, resulting in an increase in a time difference between thepoint in time of generating the video and the point in time ofdisplaying the video. Moreover, it is difficult to predict a delay by acongestion state of a network and a reception error often occurs.

The present invention has been achieved in light of such problems, andan object of the present invention is to provide an image correctiontechnology capable of correcting an image in response to a delay fromgeneration and delivery of the image to display of the image.

Solution to Problems

To solve the problems, an image correction apparatus according to onemode of the present invention includes: an acquisition section thatacquires an image rendered on the basis of a position or an orientationof a head mounted display at a first time together with informationregarding the position or the orientation at the first time; and acorrection section that corrects the image by acquiring informationregarding a position or an orientation of the head mounted display at asecond time of displaying the image, translating or rotationally movinga visual field of the head mounted display in a screen coordinate systemin response to a difference between the position or the orientation atthe first time and the position or the orientation at the second time,and pasting the image on the translated or rotationally moved visualfield as a texture.

Another mode of the present invention is an image correction method.This method includes: an acquisition step of acquiring an image renderedon the basis of a position or an orientation of a head mounted displayat a first time together with information regarding the position or theorientation at the first time; and a correction step of correcting theimage by acquiring information regarding a position or an orientation ofthe head mounted display at a second time of displaying the image,translating or rotationally moving a visual field of the head mounteddisplay in a screen coordinate system in response to a differencebetween the position or the orientation at the first time and theposition or the orientation at the second time, and pasting the image onthe translated or rotationally moved visual field as a texture.

It is noted that conversion of optional combinations of theaforementioned constituent elements and expressions of the inventionamong a method, an apparatus, a system, a computer program, a datastructure, a recording medium, and so forth is also effective as a modeof the present invention.

Advantageous Effect of Invention

According to the present invention, it is possible to correct an imagein response to a delay generated from generation and delivery of theimage to display of the image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an external view of a head mounted display.

FIG. 2 is a configuration diagram of an image generation systemaccording to an embodiment of the present invention.

FIG. 3 is a functional configuration diagram of a head mounted display.

FIG. 4 is a functional configuration diagram of an image generationapparatus according to the present embodiment.

FIG. 5 is a functional configuration diagram of an image correctionapparatus according to the present embodiment.

FIG. 6 is a flowchart for explaining an image generation procedureaccording to the present embodiment.

FIG. 7 is a detailed time chart of the image generation procedureaccording to the present embodiment.

FIGS. 8(a) to 8(c) are explanatory diagrams of rotational movement of abillboard.

FIG. 9 is a detailed explanatory diagram of the rotational movement ofthe billboard.

FIG. 10 is an explanatory diagram of projection of the rotated billboardonto a screen.

DESCRIPTION OF EMBODIMENT

FIG. 1 is an external view of a head mounted display 100. The headmounted display 100 is a display apparatus worn on a head of a user sothat the user can view a still image, a moving image, or the likedisplayed on a display and listen to audio, music, or the like outputfrom a headphone.

A gyro sensor, an acceleration sensor, or the like incorporated into orexternally attached to the head mounted display 100 can measure positioninformation regarding the head of the user wearing the head mounteddisplay 100 and posture (orientation) information such as a rotationalangle and an inclination of the head.

A camera unit is mounted on the head mounted display 100, and the cameraunit can capture an image of an outside world while the user is wearingthe head mounted display 100.

The head mounted display 100 is an example of a “wearable display.”While a method of generating an image displayed on the head mounteddisplay 100 is described herein, the image generation method accordingto the present embodiment is applicable not only to a case of user'swearing the head mounted display 100 in a narrow sense but also to casesof user's wearing glasses, a glasses display, a glasses camera,headphones, a headset (headphones with a microphone), earphones, anearring, an ear hook camera, a hat, a hat with a camera, a headband, andthe like.

FIG. 2 is a configuration diagram of an image generation systemaccording to the present embodiment. By way of example, the head mounteddisplay 100 is connected to an image correction apparatus 200 via aninterface 300 such as an HDMI (registered trademark) (High-DefinitionMultimedia Interface) that is a standard of communication interfaces fortransmitting a video and an audio in the form of digital signals.

The image correction apparatus 200 is, for example, a video gamemachine. The image correction apparatus 200 is connected to an imagegeneration apparatus 500 via a network 400. The image correctionapparatus 200 is a client and the image generation apparatus 500 is aserver. The image generation apparatus 500 may provide an onlineapplication of a game or the like in which a plurality of users canparticipate via the network, to the image correction apparatus 200.

The image correction apparatus 200 transmits position/orientationinformation regarding the head mounted display 100 to the imagegeneration apparatus 500. The image generation apparatus 500 renders animage to be displayed on the head mounted display 100 on the basis ofthe received position/orientation information regarding the head mounteddisplay 100, encodes rendering data, and transmits the encoded renderingdata to the image correction apparatus 200 as a video stream. The imagecorrection apparatus 200 decodes the received video stream, corrects therendering data to be adapted to a latest position/orientation of thehead mounted display 100, and displays the corrected rendering data onthe head mounted display 100.

FIG. 3 is a functional configuration diagram of the head mounted display100.

A control section 10 is a main processor that processes signals such asan image signal and a sensor signal, an instruction, and data and thatoutputs the processed signals, instruction, and data. An input interface20 receives an operation signal and a setting signal from a user andsupplies the operation signal and the setting signal to the controlsection 10. An output interface 30 receives the image signal from thecontrol section 10 and displays the image signal on a display panel 32.

A communication control section 40 transmits the data input from thecontrol section 10 to outside via a network adapter 42 or an antenna 44by wired or wireless communication. The communication control section 40receives data from the outside via the network adapter 42 or the antenna44 by the wired or wireless communication and outputs the data to thecontrol section 10.

A storage section 50 temporarily stores the data, parameters, theoperation signal, and the like processed by the control section 10.

An orientation sensor 64 detects position information regarding the headmounted display 100 and orientation information such as a rotationalangle and an inclination of the head mounted display 100. Theorientation sensor 64 is realized by combining a gyro sensor, anacceleration sensor, an angular acceleration sensor, and the like asappropriate. Alternatively, back and forth, right and left, and up anddown motions of the user's head may be detected using a motion sensorthat is a combination of at least one or more sensors out of athree-axis geomagnetic sensor, a three-axis acceleration sensor, and athree-axis gyro (angular velocity) sensor.

An external input/output terminal interface 70 is an interface forconnecting a peripheral device such as a USB (Universal Serial Bus)controller. An external memory 72 is an external memory such as a flashmemory.

An HDMI transmitting and receiving section 90 transmits and receivesvideo and audio digital signals to and from the image correctionapparatus 200 in accordance with the HDMI. The HDMI transmitting andreceiving section 90 receives an image generated by the image correctionapparatus 200 from the image correction apparatus 200 using an HDMIchannel, and supplies the image to the control section 10.

The control section 10 can supply an image and text data to the outputinterface 30 to display the image and the text data on the display panel32, and can supply the image and the text data to the communicationcontrol section 40 to transmit the image and the text data to theoutside.

The image correction apparatus 200 is notified of currentposition/orientation information regarding the head mounted display 100detected by the orientation sensor 64 via either the communicationcontrol section 40 or the external input/output terminal interface 70.Alternatively, the HDMI transmitting and receiving section 90 maytransmit the current position/orientation information regarding the headmounted display 100 to the image correction apparatus 200.

FIG. 4 is a functional configuration diagram of the image generationapparatus 500 according to the present embodiment. FIG. 4 is a viewdepicting blocks paying attention to functions, and these functionalblocks can be realized in various forms by hardware only, software only,and a combination of the hardware and the software.

At least partial functions of the image generation apparatus 500 may beimplemented in the image correction apparatus 200.

A transmitting and receiving section 560 receives current (referred toas “first time”) position/orientation information regarding the headmounted display 100 from the image correction apparatus 200, andsupplies the position/orientation information to a position/orientationacquisition section 510.

A viewpoint/line-of-sight setting section 520 sets a viewpoint locationand a line-of-sight direction of the user using the position/orientationinformation regarding the head mounted display 100 acquired by theposition/orientation acquisition section 510.

A rendering section 530 renders an object in a virtual space viewed froma viewpoint location of the user wearing the head mounted display 100 ina line-of-sight direction in accordance with the viewpoint location andthe line-of-sight direction of the user set by theviewpoint/line-of-sight setting section 520, and stores the object inthe virtual space in an image storage section 540.

An encoding section 550 encodes a rendering result stored in the imagestorage section 540, and applies the encoded video stream to thetransmitting and receiving section 560.

The transmitting and receiving section 560 receives the first timeposition/orientation information regarding the head mounted display 100used in rendering by the rendering section 530 from theposition/orientation acquisition section 510, and transmits video streamdata together with the first time position/orientation information tothe image correction apparatus 200. The video stream in which video dataand the position/orientation information are associated per frame isthereby delivered from the image generation apparatus 500 that is theserver to the image correction apparatus 200 that is the client.

FIG. 5 is a functional configuration diagram of the image correctionapparatus 200 according to the present embodiment. FIG. 5 is a viewdepicting blocks paying attention to functions, and these functionalblocks can be realized in various forms by hardware only, software only,and a combination of the hardware and the software.

At least partial functions of the image correction apparatus 200 may beimplemented in the control section 10 in the head mounted display 100.

A transmitting and receiving section 210 receives the video stream dataand the first time position/orientation information regarding the headmounted display 100 used in rendering from the image generationapparatus 500 per frame. The transmitting and receiving section 210applies the received video stream data to a decoding section 220, andapplies the first time position/orientation information regarding thehead mounted display 100 to a position/orientation acquisition section230.

The decoding section 220 decodes the video stream data and stores adecoded image together with the first time position/orientationinformation in an image storage section 250.

An HDMI transmitting and receiving section 270 receives latest (referredto as “second time”) position/orientation information from the headmounted display 100 and applies the second time position/orientationinformation to the position/orientation acquisition section 230.

The position/orientation acquisition section 230 applies the first timeposition/orientation information and the second timeposition/orientation information to a correction section 260.

The correction section 260 reads the decoded image from the imagestorage section 250, performs a reprojection process for correcting theimage data in response to a difference between the first timeposition/orientation information and the second timeposition/orientation information, and converts the decoded image into animage viewed at the latest viewpoint location of the head mounteddisplay 100 in the line-of-sight direction. The correction section 260applies corrected image data to the HDMI transmitting and receivingsection 270, and the HDMI transmitting and receiving section 270transmits the corrected image data to the head mounted display 100.

An error detection section 240 detects, as an error, a case in which thevideo stream data delays due to network congestion or the like and inwhich the transmitting and receiving section 210 is unable to receivenecessary video stream data at timing of frame rendering. The errordetection section 240 notifies the correction section 260 of occurrenceof the error. In a case of occurrence of the error due to the delay, thecorrection section 260 reads a previous frame of the decoded image fromthe image storage section 250, performs the reprojection process forcorrecting the previous frame of the decoded image in response to adifference between past time position/orientation information used inrendering of the previous frame of the decoded image and the second timeposition/orientation information, and converts the decoded image into animage viewed at the latest viewpoint location of the head mounteddisplay 100 in the line-of-sight direction.

Reprojection will now be described. In a case in which the head mounteddisplay 100 takes on a head tracking function and a virtual realityvideo is generated by changing the viewpoint and the line-of-sightdirection to be interlocked with the motion of the user's head, a delayis generated from generation and display of the virtual reality video.As a result, a lag is generated between the direction of the user's headpremised the point in time of generating the video and the direction ofthe user's head at the point in time of displaying the video on the headmounted display 100, and the user often falls into the sense of feelingsick (referred to as “VR Sickness (Virtual Realty Sickness)” or thelike).

In such a way, it takes long time to detect a motion of the head mounteddisplay 100, issue a rendering command from a CPU (Central ProcessingUnit), execute rendering by a GPU (Graphics Processing Unit), and outputa rendered image to the head mounted display 100. It is assumed, forexample, that rendering is executed at a frame rate of 60 fps(frame/second) and that a delay of one frame is generated from detectionof the motion of the head mounted display 100 is detected to output ofthe image. This delay is approximately 16.67 milliseconds at the framerate of 60 fps and sufficient for a person to perceive a lag.

Furthermore, if the server encodes rendered image data and transmits theimage data to the client via the network, communication time is added tothe time taken for rendering and a further delay is, therefore,generated until the image data is displayed on the head mounted display100.

To address the problem, a process referred to as “timewarp” or“reprojection” is performed to correct the rendered image to be adaptedto the latest position and the latest orientation of the head mounteddisplay 100 so that it is difficult for a person to perceive the lag.

FIG. 6 is a flowchart for explaining an image generation procedureaccording to the present embodiment.

The transmitting and receiving section 210 in the image correctionapparatus 200 that is the client transmits the currentposition/orientation information regarding the head mounted display 100to the image generation apparatus 500 that is the server (S10).

In the image generation apparatus 500, the transmitting and receivingsection 560 receives the current position/orientation informationregarding the head mounted display 100 from the image correctionapparatus 200 (S12). The rendering section 530 renders an image to bedisplayed on the head mounted display 100 on the basis of the currentposition/orientation information regarding the head mounted display 100(S14).

The encoding section 550 encodes image data obtained as a result ofrendering (S16). The transmitting and receiving section 560 transmits anencoded video stream together with the position/orientation informationregarding the head mounted display 100 used in rendering to the imagecorrection apparatus 200 (S18).

In the image correction apparatus 200, the transmitting and receivingsection 210 receives the video stream and the position/orientationinformation regarding the head mounted display 100 used in renderingfrom the image generation apparatus 500 (S20). The decoding section 220decodes the video stream (S22). The correction section 260 executes thereprojection process for correcting the rendering data to be adapted tothe latest position/orientation of the head mounted display 100 (S24).The HDMI transmitting and receiving section 270 transmits the correctedimage data to the head mounted display 100 to display the image data onthe head mounted display 100 (S26).

A round trip time (RTT) delay is generated since the image correctionapparatus 200 transmits the position/orientation information regardingthe head mounted display 100 to the image generation apparatus 500 instep S10 until the image is displayed on the head mounted display 100.The reprojection process in step S24 interpolates the image so that theuser does not feel this delay.

FIG. 7 is a detailed time chart of the image generation procedureaccording to the present embodiment.

At time T1, the image correction apparatus 200 that is the clientacquires the position/orientation information regarding the head mounteddisplay 100 and transmits the position/orientation information to theimage generation apparatus 500 that is the server.

At time T2, the image generation apparatus 500 renders an image to bedisplayed on the head mounted display 100 in response to the receivedposition/orientation information regarding the head mounted display 100.At time T3, the image generation apparatus 500 encodes the renderedimage, and transmits the encoded video stream data together with theposition/orientation information regarding the head mounted display 100used in rendering to the image correction apparatus 200.

At time T4, the image correction apparatus 200 decodes the receivedvideo stream. At time T5, the image correction apparatus 200 performsthe reprojection process on the decoded image. The decoded image isrendered on the basis of the position/orientation information regardingthe head mounted display 100 at the time T1. At the time T5, the userwearing the head mounted display 100 rotates the head by an angle θ1,and the decoded image is interpolated in response to a change in theposition/orientation information regarding the head mounted display 100for a difference ΔT=T5−T1. At time T6, the image correction apparatus200 displays the reprojection-processed decoded image on the headmounted display 100.

At time T7, the image correction apparatus 200 acquires theposition/orientation information regarding the head mounted display 100and transmits the position/orientation information to the imagegeneration apparatus 500.

At time T8, the image generation apparatus 500 renders an image to bedisplayed on the head mounted display 100 in response to the receivedposition/orientation information regarding the head mounted display 100.At time T9, the image generation apparatus 500 encodes the renderedimage and transmits the encoded video stream data together with theposition/orientation information regarding the head mounted display 100used in rendering to the image correction apparatus 200; however, adelay caused by network congestion is generated and the video streamdata is late for next frame timing.

At time T10, the image correction apparatus 200 detects occurrence of anerror in data reception, and reads the decoded image at the time T4 thatis a previous frame (referred to as “previous decoded image”) from amemory. At time T11, the image correction apparatus 200 performs thereprojection process on the previous decoded image. The previous decodedimage is rendered on the basis of the position/orientation informationregarding the head mounted display 100 at the time T1. At the time T11,the user wearing the head mounted display 100 rotates the head by anangle θ2, and the image is interpolated in response to a change in theposition/orientation information regarding the head mounted display 100for a difference ΔT=T11−T1. At time T12, the image correction apparatus200 displays the reprojection-processed image on the head mounteddisplay 100.

The reprojection process performed by the correction section 260 in theimage correction apparatus 200 will be described in detail withreference to FIGS. 8 to 10. In the present embodiment, the clientreceives the image already rendered by the server and interpolates theimage; thus, the present embodiment uses a method of interpolating theimage by as much as a movement of a visual field by rotating a billboardwithout performing pixel-by-pixel reprojection. Since linearinterpolation of the image is performed by converting coordinate valuesof four vertexes in screen coordinates, the method can easily beimplemented using a linear interpolation function of a GPU of theclient.

FIGS. 8(a) to 8(c) are explanatory diagrams of rotational movement ofthe billboard.

FIG. 8(a) depicts a first visual field 600 when the user wearing thehead mounted display 100 is facing the front and a second visual field610 when the user rotates the head to the right by an angle of θ. Whilethe server renders an image on the first visual field 600 at the firsttime at which the user is facing the front, timing of displaying therendered image is the second time at which the user rotates the head bythe angle of θ. At the second time, it is necessary to correct therendered image and display the image on the second visual field 610 thatis a plane perpendicular to a direction of the angle θ.

FIG. 8(b) depicts a method of rotating the billboard to interpolaterotational movement of the visual field by the angle θ in FIG. 8(a). Thebillboard in the front first visual field 600 is rotated in an oppositedirection by the angle θ, and the image rendered on the first visualfield 600 is pasted on a rotated billboard 620 as a texture. The imagerendered on the first visual field 600 is thereby linearly interpolatedand displayed on the second visual field 610 of FIG. 8(a).

FIG. 8(c) depicts an interpolated image displayed on the head mounteddisplay 100 when the user rotates the head to the right by the angle θ.The image rendered on the first visual field 600 is linearlyinterpolated and displayed on the second visual field 610.

The rotational movement of the billboard of FIG. 8(b) is made byinversely rotating four vertexes of the first visual field 600 by theangle θ. x and y coordinates of the four vertexes are screencoordinates, z coordinate is a distance d from a rotation center, and avalue calculated from a viewing angle Fov is used as the distance d.

It is noted that in a case in which the user wearing the head mounteddisplay 100 translates the head, the billboard of the first visual field600 may be translated in the opposite direction and the texture on thefirst visual field 600 may be pasted.

FIG. 9 is a detailed explanatory diagram of the rotational movement ofthe billboard.

Screen coordinates of four vertexes P₁, P₂, P₃, and P₄ of the billboardin the visual field are (−1, −1), (−1, 1), (1, −1), and (1, 1), andrepresented as (−1, −1, d), (−1, 1, d), (1, −1, d), and (1, 1, d),respectively in a coordinate system extended in three dimensions usingthe distance d from a viewpoint to a screen as the z coordinate.

The distance d from the viewpoint to the screen is calculated by thefollowing equations using left and right viewing angles Fov_(left) andFov_(right).

a=2× Fov_(left)/(Fov_(left)+ Fov_(right))

b=2× Fov_(right)/(Fov_(left)+ Fov_(right))

d=b/tan(Fov_(right))

The four vertexes P₁, P₂, P₃, and P₄ of the billboard are rotatedinversely by the angle θ in a three-dimensional space, and coordinatevalues of rotated four vertexes P₁′, P₂′, P₃′, and P₄′ of the billboardare calculated. If an orientation of the head mounted display 100 isrepresented by quaternions, a rotational angle is calculated as adifference between the orientation of the head mounted display 100 onthe first visual field 600 and the orientation of the head mounteddisplay 100 on the second visual field 610.

A method of calculating three-dimensional rotations or the like relatedto computer graphics using quaternions is described in “Introduction toQuaternions for 3D-CG programmers” (Kohgaku-Sha Co., Ltd., January2004).

The orientations q_(A) and q_(B) of the head mounted display 100 on thefirst visual field 600 and the second visual field 610 are representedby the following equations.q _(A)=(qx _(A) ,qy _(A) , qz _(A) ,qw _(A))q _(B)=(qx _(B) ,qy _(B) ,qz _(B) ,qw _(B))

The rotational angle q_(AB) of the head mounted display 100 is obtainedby calculating a difference between the orientations q_(A) and q_(B) ofthe head mounted display 100 on the first visual field 600 and thesecond visual field 610 by the following equation.q _(AB) =q _(A) *q _(B) ⁻¹

A reciprocal of the quaternion q_(B) ⁻¹ is as follows.q _(B) ⁻¹=(−qx _(B) ,−qy _(B) ,−qz _(B) ,qw _(B))The rotational angle q_(AB) is, therefore, obtained by the followingequation.q _(AB) =q _(A) *q _(B) ⁻¹=(qx _(A) ,qy _(A) ,qz _(A) ,qw _(A))*(−qx _(B) ,−qy _(B) ,−qz _(B) ,qw_(B))=(qw _(A) *qx _(B) +qx _(A) *qw _(B) +qy _(A)*(−qz _(B))−qz _(A)*(−qy_(B)),qw _(A) *qy _(B) −qx _(A)*(−qz _(B))+qy _(A) *qw _(B) +qz _(A)*(−qx_(B)),qw _(A) *qz _(B) +qx _(A)*(−qy _(B))−qy _(A)*(−qx _(B))+qz _(A) *qw_(B),qw _(A) *qx _(B) −qx _(A)*(−qy _(B))−qy _(A)*(−qy _(B))−qz _(A)*(−qz_(B)))

By rotating the four vertexes P₁, P₂, P₃, and P₄ of the billboard by therotational angle q_(AB), coordinate values of P₁′, P₂′, P₃′, and P₄′ areobtained. To rotate the four vertexes in quaternions, the coordinatevalues of the four vertexes are represented by four-dimensional vectorsas follows.P ₁=(−1,−1,d,0)P ₂=(−1,1,d,0)P ₃=(1,−1,d,0)P ₄=(1,1,d,0)

Since rotation of a point p by a quaternion q is given by q⁻¹*p*q, therotated four vertexes P₁′, P₂′, P₃′, and P₄′ are obtained by thefollowing equation. In the following equation, n is any of 1 to 4. pxand py are screen coordinates and given as either 1 or −1 as describedabove. pz is the distance d from the viewpoint to the screen.Furthermore, the rotational angle is defined as q_(AB)=(qx _(AB) ,qy _(AB) ,qz _(AB) ,qw _(AB)).Pn′=q _(AB) ⁻¹ *Pn*q _(AB)=(−qx _(AB) ,−qy _(AB) ,−qz _(AB) ,qw _(AB))*(px,py,pz,0)*(qx _(AB) ,qy_(AB) ,qz _(AB) ,qw _(AB))=(qw _(AB) *px+(−qy _(AB))*pz−(−qz _(AB))*py,qw _(AB) *py−(−qx _(AB))*pz+(−qz _(AB))*px,qw _(AB) *pz+(−qx _(AB))*py−(−qy _(AB))*px,−(−qx _(AB))*px−(−qy _(AB))*py−(−qz _(AB))*pz)*(qx _(AB) ,qy _(AB) ,qz_(AB) ,qw _(AB))=((qx _(AB) ² −qy _(AB) ² −qz _(AB) ² +qw _(AB) ²)px+2(qx _(AB) *qy_(AB) +qz _(AB) *qw _(AB))py+2(qx _(AB) *qz _(AB) −qy _(AB) *qw_(AB))pz,2(qx _(AB*) qy _(AB) *−qz _(AB) *qw _(AB))px−(qx _(AB) ² −qy _(AB) ² +qz_(AB) ² −qw _(AB) ²)py+2(qy _(AB) *qz _(AB) +qx _(AB) qw _(AB))pz,2(qx _(AB) *qz _(AB) +qy _(AB) +qy _(AB) *qw _(AB))px+2(qy _(AB) *qz_(AB) −qx _(AB) *qw _(AB))py−(qx _(AB) ² +gy _(AB) ² −qz _(AB) ² −qw_(AB) ²)pz,−2*qy _(AB) *qz _(AB) *px _(AB))

FIG. 10 is an explanatory diagram of projection of the rotated billboardonto the screen.

Since the rotated four vertexes P₁′, P₂′, P₃′, and P₄′ of the billboardare present in the three-dimensional space, it is necessary to projectagain the four vertexes P₁′, P₂′, P₃′, and P₄′ onto the screen. Asrepresented by the following equation, coordinate values of fourvertexes Pn′_(screen) on the screen are obtained by multiplying a ratioof the distance d from the viewpoint to the screen to a depth pz′ of therotated billboard by the coordinate values px′ and py′ of the fourvertexes Pn′ of the rotated billboard. In the equation, n is any of 1 to4.Pn′ _(screen)=(px′,py′)*d/pz′=(px′*d/pz′,py′*d/pz′)

The present invention has been described so far. An ordinaryreprojection process is performed to compensate for processing time fromrendering to display, and absorbs a time difference from a point in timeof image generation to a point in time of image display by correcting animage generated on the premise of the viewpoint location and theline-of-sight direction at the point in time of image generation usingthe viewpoint location and the line-of-sight direction at the point intime of image display. By contrast, in a case of streaming delivery ofimages to the head mounted display 100 via the network as in the presentembodiment, the reprojection process is performed to absorb a timedifference including the communication time.

In the case of the streaming delivery, the client transmits the currentposition/orientation information regarding the head mounted display 100to the server, and the server renders an image on the basis of thereceived position/orientation information and transmits the video streamtogether with the position/orientation information used at the time ofrendering to the client. The client performs the reprojection process onthe image rendered by the server to compensate the difference betweenthe position/orientation information used at the time of rendering andthe latest position/orientation information. It is thereby possible toabsorb the time difference from the point in time at which the clientrequests the server to perform rendering to the communicationtime-inclusive point in time of displaying the image.

Furthermore, in the case of the streaming delivery, a communicationdelay occurs due to the network congestion or the like and an error thatthe video stream does not arrive on time possibly occurs. In the case ofoccurrence of the data arrival error, it is possible to flexibly dealwith the communication delay by reusing the previous decoded image andperforming the reprojection process on the previous decoded image to beadapted to the latest position/orientation of the head mounted display100.

The present invention has been described so far with reference to theembodiment. The embodiment is given as an example, and a person skilledin the art would understand that various modifications can be made forcombinations of the constituent elements and the processes and that suchmodifications fall within the scope of the present invention. Suchmodifications will be described.

REFERENCE SIGNS LIST

10 Control section, 20 Input interface, 30 Output interface, 32 Displaypanel, 40 Communication control section, 42 Network adapter, 44 Antenna,50 Storage section, 64 Orientation sensor, 70 External input/outputterminal interface, 72 External memory, 90 HDMI transmitting andreceiving section, 100 Head mounted display, 200 Image correctionapparatus, 210 Transmitting and receiving section, 220 Decoding section,230 Position/orientation acquisition section, 240 Error detectionsection, 250 Image storage section, 260 Correction section, 270 HDMItransmitting and receiving section, 300 Interface, 400 Network, 500Image generation apparatus, 510 Position/orientation acquisitionsection, 520 Viewpoint/line-of-sight setting section, 530 Renderingsection, 540 Image storage section, 550 Encoding section, 560Transmitting and receiving section.

INDUSTRIAL APPLICABILITY

The present invention is applicable to an image correcting technology.

The invention claimed is:
 1. An image correction apparatus comprising:an acquisition section that acquires an image rendered on a basis of aposition or an orientation of a head mounted display at a first timetogether with information regarding the position or the orientation atthe first time; and a correction section that corrects the image byacquiring information regarding a position or an orientation of the headmounted display at a second time of displaying the image, translating orrotationally moving a visual field of the head mounted display in ascreen coordinate system in response to a difference between theposition or the orientation at the first time and the position or theorientation at the second time, and pasting the image on the translatedor rotationally moved visual field as a texture; and a storage sectionthat retains a decoded image at previous time obtained by decoding anencoded stream which is obtained by encoding the image received via anetwork together with information regarding a position or an orientationof the head mounted display at the previous time, wherein in a case inwhich an error in receiving the encoded stream is detected, thecorrection section corrects the decoded image at the previous time bytranslating or rotationally moving the visual field in the screencoordinate system in response to a difference between the position orthe orientation at the previous time and the position or the orientationat the second time, and pasting the decoded image at the previous timeon the translated or rotationally moved visual field as the texture. 2.An image correction method comprising: acquiring an image rendered on abasis of a position or an orientation of a head mounted display at afirst time together with information regarding the position or theorientation at the first time; and correcting the image by acquiringinformation regarding a position or an orientation of the head mounteddisplay at a second time of displaying the image, translating orrotationally moving a visual field of the head mounted display in ascreen coordinate system in response to a difference between theposition or the orientation at the first time and the position or theorientation at the second time, and pasting the image on the translatedor rotationally moved visual field as a texture; and retaining a decodedimage at previous time obtained by decoding an encoded stream which isobtained by encoding the image received via a network together withinformation regarding a position or an orientation of the head mounteddisplay at the previous time, wherein in a case in which an error inreceiving the encoded stream is detected, the correcting includescorrecting the decoded image at the previous time by translating orrotationally moving the visual field in the screen coordinate system inresponse to a difference between the position or the orientation at theprevious time and the position or the orientation at the second time,and pasting the decoded image at the previous time on the translated orrotationally moved visual field as the texture.
 3. A non-transitory,computer readable storage medium containing a computer program, whichwhen executed by a computer, causes the computer to carry out actions,comprising: acquiring an image rendered on a basis of a position or anorientation of a head mounted display at a first time together withinformation regarding the position or the orientation at the first time;and correcting the image by acquiring information regarding a positionor an orientation of the head mounted display at a second time ofdisplaying the image, translating or rotationally moving a visual fieldof the head mounted display in a screen coordinate system in response toa difference between the position or the orientation at the first timeand the position or the orientation at the second time, and pasting theimage on the translated or rotationally moved visual field as a texture;and retaining a decoded image at previous time obtained by decoding anencoded stream which is obtained by encoding the image received via anetwork together with information regarding a position or an orientationof the head mounted display at the previous time, wherein in a case inwhich an error in receiving the encoded stream is detected, thecorrecting includes correcting the decoded image at the previous time bytranslating or rotationally moving the visual field in the screencoordinate system in response to a difference between the position orthe orientation at the previous time and the position or the orientationat the second time, and pasting the decoded image at the previous timeon the translated or rotationally moved visual field as the texture.